Beating the 'world champion' evolutionary algorithm via REVAC tuning

We present a case study demonstrating that using the REVAC parameter tuning method we can greatly improve the ‘world champion’ EA (the winner of the CEC2005 competition) with little effort. For ‘normal’ EAs the margins for possible improvements are likely much bigger. Thus, the main message of this paper is that using REVAC great performance improvements are possible for many EAs at moderate costs. Our experiments also disclose the existence of ‘specialized generalists’, that is, EAs that are generally good on a set of test problems, but only w.r.t. one performance measure and not along another one. This shows that the notion of robust parameters is questionable and the issue requires further research. Finally, the results raise the question what the outcome of the CEC-2005 competition would have been, if all of EAs had been tuned by REVAC, but without further research it remains an open question whether we crowned the wrong king. I. BACKGROUND AND OBJECTIVES Finding appropriate parameter values for evolutionary algorithms (EA) is one of the persisting grand challenges of the evolutionary computing (EC) field. As explained by Eiben et al. in [7] this challenge can be addressed before the run of the given EA (parameter tuning) or during the run (parameter control). In this paper we focus on parameter tuning, that is, we are seeking good parameter values off-line and use these values for the whole EA run. In today’s practice, this tuning problem is usually ‘solved’ by conventions (mutation rate should be low), ad hoc choices (why not use uniform crossover), and experimental comparisons on a limited scale (testing combinations of three different crossover rates and three different mutation rates). Until recently, there were not many workable alternatives. However, by the developments over last couple of years now there are a number of tuning methods and corresponding software packages that enable EA practitioners to perform tuning without much effort. In particular, REVAC [16], [15], [18] and SPOT [3], [5], [4] are well developed and documented. Using algorithmic parameter tuners for EAs offers benefits on different time scales. The immediate benefits are obtained by the improved EA performance. Here the gains can be substantial, while the costs are low. In particular, the tuned EA can greatly outperform the EA based on usual parameter values, while the costs of a tuning session are by all means acceptable, typically in the range of hours. This makes algorithmic parameter tuners interesting for practitioners as well as EC scientists engaged in a performance-based competition (implicitly over a sequence of publications, or explicitly within a programming contest). The long term promise of Vrije Universiteit Amsterdam, The Netherlands, {sksmit, gusz}@cs.vu.nl good and cheap tuning methods come from the accumulated information about the relationship between parameter values and EA performance. For instance, we might learn that parameter x has almost no impact on performance, that parameter x and y have a strong correlation, or that a certain EA will work better if we turn a constant c in its code into a variable v and tune it. Over the years, such information can be generalized into new insights and knowledge about EA behavior, leading to a deeper understanding of evolutionary computing in general. The main objective of this paper is to demonstrate the fist type of benefit. (Clearly, this also contributes to the long term benefits for the whole field by obtaining and publishing information about parameter values and EA performance.) To this end, we carry out an experimental comparison by the usual EC template: “Our EA beats your EA on an interesting set of test functions”, where the only difference between “our EA” and “your EA” is that “our EA” is simply “your EA” with tuned parameter values. To make the demonstration convincing we use an EA that has proved to be very good, hence hard to improve. To find such an EA we turn to the CEC-2005 contest on real valued function optimization, take the overall winner (G-CMA-ES) and try to improve its performance over the whole test suite by tuning it with REVAC. II. PARAMETERS, TUNERS, AND UTILITY LANDSCAPES To obtain a detailed view on parameter tuning we distinguish three layers: the application layer, the algorithm layer, and the design or tuning layer, see Figure 1. Fig. 1. The three main layers in the hierarchy of parameter tuning. As this figure indicates, the whole scheme can be divided into two optimization problems. The lower part of this threetier hierarchy consists of a problem on the application layer (e.g., the traveling salesman problem) and an EA (e.g., a genetic algorithm) on the algorithm layer trying to find an